Approximate Faithful Embedding in Learning

In this paper we consider the problem of embedding the input and hypotheses of boolean function classes in other classes, such that the natural metric structure of the two spaces is approximately preserved. We rst prove some general properties of such embedding and then suggest and discuss possible approximate embedding in the class of \half-spaces" (single layer perceptrons) with dimension polynomial in the VC dimension of the original problem. Our main result is that such an approximate embedding by half-spaces is possible for a class of problems, which we call \informative". These are homogeneous learnable concept classes for which the dual problem (learning the input from labels of random hypotheses) has a similar VC-dimension, and the variance of the generalization error is bounded from zero. This is a distribution dependent property which we need to hold for most distributions on instances and hypotheses. We argue that many important learning classes are \informative" for typical distributions, e.g. geometric concepts, neural networks, decision trees, etc.